Experiments with Reinforcement Learning in Problems with Continuous State and Action Spaces

A key element in the solution of reinforcement learning problems is the value function The purpose of this function is to measure the long term utility or value of any given state The function is important because an agent can use this measure to decide what to do next A common problem in reinforcement learning when applied to systems having continuous states and action spaces is that the value function must operate with a domain consisting of real valued variables which means that it should be able to represent the value of in nitely many state and action pairs For this reason function approximators are used to represent the value function when a close form solution of the optimal policy is not available In this paper we extend a previously proposed reinforcement learning algorithm so that it can be used with function approximators that generalize the value of individual experiences across both state and action spaces In particular we discuss the bene ts of using sparse coarse coded function approximators to represent value functions and describe in detail three implementations CMAC instance based and case based Additionally we discuss how function approximators having di erent degrees of resolution in di erent regions of the state and action spaces may in uence the performance and learning e ciency of the agent We propose a simple and modular technique that can be used to implement function approximators with non uniform degrees of resolution so that it can represent the value function with higher accuracy in important regions of the state and action spaces We performed extensive experiments in the double integrator and pendulum swing up systems to demonstrate the proposed ideas Kewords Reinforcement learning function approximation memory based methods continu ous domains optimal control resource preallocation